RS-YABI: A workflow system for Remote Sensing Processing in AusCover

Earth observation from space involves accessing, sharing, and processing huge volumes of data collected from satellite sensors. It is crucial for users to easily analyze and process the datasets, which may require complex computation on very high volume datasets, particularly for large-scale spatio-temporal analysis. Generation of data products, access to, analysis and visualization of these large datasets is often perceived as a highly technical and daunting set of complex steps. In this context, a workflow application can help users to manage and process large volumes of satellite data and execute scientific experiments on distributed resources. This paper reports our work on utilizing a workflow engine (RS-YABI) that works with data from Moderate Resolution Imaging Spectroradiometer (MODIS) sensors and enables researchers to process datasets using High Performance Computing (HPC) resources. The benefits of this approach include abstracting the complexity of the underlying processing environment and easy processing of raw satellite data to generate derived data products in standard formats. We start with providing an overview of AusCover, a facility of the Terrestrial Ecosystem Research Network (TERN), as the application context of our work. It provides a national expert network and a data delivery service for provision of Australian biophysical remote sensing data time-series, continental-scale map products, and selected high-resolution datasets over Terrestrial sites. AusCover supports a nationally consistent approach to the delivery and calibration/validation of key current and future core satellite-derived datasets. Some background of workflows and remote sensing data processing is described next. In our context, using a workflow for processing remote sensing data offers several advantages, such as (a) the ability to design an operational process by leveraging existing application modules, (d) utilizing distributed resources to increase throughput or reduce execution costs, (c) obtaining specific processing capabilities as required by users, and (d) hiding technical complexity behind a straightforward user interface. It is followed by the description of YABI, a workflow engine developed at Murdoch University, Australia, originally for use in a bio-informatics context supporting linear workflows similar to those used in remote sensing processing. It provides a flexible mechanism for joining independent executables through wrappers that can be implemented in virtually any language. Examples of ongoing work on the development of wrapping scripts for customizing specific remote sensing operations, orchestrating multiple modules and simplifying user input choices are described. One of YABI’s key features is its abstraction of the backend HPC resources such as file stores and execution engines which makes it ideal for use in a distributed data system such as that developed by AusCover. YABI has been deployed as RS-YABI (for Remote Sensing) on HPC resources in the National Computational Infrastructure (NCI). The RS-YABI instance is deployed on a Virtual Machine (VM), which provides a self-contained environment independent of the HPC system. Multiple workflows for processing MODIS sensor data have already been successfully developed. We also present two brief case studies to demonstrate the applicability of RS-YABI in practical application areas such as dust detection, surface temperatures (land and ocean) monitoring, and smoke plume detection. We conclude the paper with a list of future research directions.